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Abstract 

In  this  report,  we  describe  a  computer-based  training  tutor,  the  D2P/Moving  Target  Tutor  (MTT)  and  our  experience 
and  results  of  running  an  experimental  study  using  undergraduate  students  (N=27)  to  evaluate  the  effectiveness  of 
D2P/MTT.  D2P/MTT  is  designed  and  created  to  train  users  to  judge  point  of  aim  quickly  to  shoot  a  moving  target. 
The  structure  of  the  tutor  is  based  on  the  Declarative  to  Procedural  (D2P)  learning  theory  that  prescribes  how 
repetitive  practice  can  lead  to  declarative  knowledge  being  proceduralized  and  how  proceduralized  skills  can  be 
retrieved  quicker  and  less  prone  to  forgetting.  We  analyzed  our  results  and  found  that  the  accuracy  rate  on  judging 
point  of  aim  of  a  moving  object  improved  from  20%  (standard  deviation  =  15%)  to  47%  (standard  deviation  =  18%). 
This  was  a  reliable  difference,  despite  lost  data.  We  also  describe  the  possible  reason  for  high  data  loss  rate  (63%) 
for  future  improvements  in  running  studies  like  this. 
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1  Introduction 

The  Applied  Cognitive  Science  Lab  at  the  Pennsylvania  State  University  derived  a  learning  theory  named 
Declarative  to  Procedure  (D2P)  from  a  series  of  studies  related  to  learning  and  training  under  different  situations  and 
schedules(Kim,  Ritter,  &  Koubek,  in  press).  A  detail  explanation  of  one  of  the  learning  theories  used  in  the 
summary  theory,  in  particular  the  declarative  and  procedural  stages  can  be  found,  for  example,  in  Anderson’s  paper 
(Anderson,  1982).  The  D2P  theory,  in  short,  describes  the  relationship  between  cognitive  knowledge  and  training 
schedule;  it  also  describes  knowledge  retention  after  its  acquisition. 

This  manuscript  is  the  result  of  an  experiment  on  a  computer-based  learning  tutor  called  the  D2P/Moving  Target 
Tutor  (MTT)  that  is  created  based  on  D2P  to  train  users  to  retrieve  knowledge  quickly.  We  show  the  effectiveness  of 
learning  declarative  knowledge  after  using  MTT  for  approximately  five  hours  with  no  other  instructional 
interventions. 

The  structure  of  the  report  is  organized  as  the  following.  First  we  briefly  describe  the  theory  behind  D2P/MTT  and 
the  structure  of  MTT.  We  then  describe  the  experiment  including  the  participants,  the  apparatus,  and  the  research 
design  and  procedure.  In  the  results  section,  we  present  how  the  students  learned  and  note  that  the  results  are 
significant  and  reliable.  In  the  end  we  discuss  our  experience  and  present  our  conclusions  including  suggestions  for 
improving  studies  like  this. 


2  The  D2P/Moving  Target  Tutor  (MTT) 

D2P/MTT  is  a  computer-based  tutor  that  teaches  learners  to  retrieve  the  Point  of  Aim  (POA)  for  a  moving  object. 
POA  is  the  result  of  combining  the  range,  speed,  and  angle  of  a  moving  object  related  to  the  observer.  Judging  POA 
is  an  important  skill  for  any  shooter,  especially  men  and  women  in  the  military,  who  are  often  firing  at  moving 
targets.  MTT  contains  roughly  four  to  five  hours  of  learning  materials,  including  text,  still  images,  audios,  videos, 
and  quizzes.  The  ultimate  goal  of  MTT  is  so  that  a  learner  would  be  able  to  retrieve  POA  quickly  after  MTT  training 
without  having  to  retrieve  each  factor  (range,  speed,  and  angel)  individually.  That  is,  learners  will  use  D2P/MTT  to 
practice  and  eventually  transform  factual  knowledge  that  later  become  part  of  their  procedural  knowledge,  moving 
knowledge  from  declarative  knowledge  to  procedural  knowledge. 

MTT  is  created  in  the  ACS  Lab  at  Penn  State  using  the  Java  programming  language.  The  instruction  portion  of  MTT 
is  constructed  in  extensible  Markup  Language  (XML)  format  to  separate  the  tutor  program/interpreter  from  the 
instructional  materials.  This  approach  allows  programmers  and  instructional  designers  to  work  separately  and 
concurrently,  and  to  make  it  easier  to  create  tutors.  This  separation  makes  it  possible  and  easier  to  extend  the  type  of 
instructional  pages  supported  by  MTT  without  changing  the  previous  tutors.  It  also  allows  us,  if  it  arises  in  the 
future,  to  change  the  tutor  interpreter  but  reuse  the  instructional  materials.  Details  about  the  type  of  pages  and  their 
specification  are  covered  in  a  separate  report  (Hiam,  2012). 

The  main  function  of  the  D2P  tutor  interpreter  is  to  read  the  XML  page  file  that  defines  the  tutor’s  pages  and  to 
render  them.  The  D2P  engine  handles  page  layout  and  switching  between  pages.  In  addition,  the  interpreter  attempts 
to  connect  to  a  server  to  log  user  data,  which  will  be  used  to  create  user  models  later.  When  there  is  no  Internet 
connection  available,  the  D2P  engine  will  save  user  data  locally  that  can  be  sent  through  email  when  the  tutor  is 
reconnected  to  the  Internet. 

On  the  instructional  material  part,  D2P/MTT  currently  has  six  sections:  pre-qualification  exercise,  moving  target 
assessment,  POA  technique,  moving  target  engagement  techniques,  qualification  exam,  and  practice  module  (see 
Figure  1  left)  and  users  can  see  sections  as  sub  sections  (Figure  1  right).  Users  will  go  through  a  series  of  training, 
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from  simple  to  complex.  Training  pages  may  contain  text,  still  images,  audio  and  video  clips,  and  quizzes,  and 
D2P/MTT  uses  all  of  these.  Figure  2  shows  an  example  of  an  exercise  where  the  user  has  to  answer  about  range 
(distance),  angle,  speed,  and  POA.  Exercises  can  be  configured  with  and  without  time  constrain. 


Menu  Page 

Click  the  target  to  enter  each  section. 


H 

H 

II 


1)  Pre-Quallflcatlon  Exercise 


3)  PoInts-of-Alm  Technique 


S)  Qualification  Exam 


2)  Moving  Target  Assessment 


4)  Moving  Target  Engagement  Techniques 


6)  Practice  Module 


Moving  Target  Engagement  Techniques 

Click  the  target  to  enter  each  section. 


1)  Observation  Techniques 


3)  Engagement  Techniques 


on  techniques  larger  lypes  ana  engagements  techniques 


2)  Moving  Target  Types 


Figure  1.  The  MTT  menu  page  (left),  and  an  MTT  sub  section  under 
Moving  Target  Engagement  Techniques  section  (right). 


Figure  2.  An  example  MTT  exercise  page. 


3  Method 

We  tested  how  well  D2P/MTT  teaches  points  of  aim  with  a  small  study  at  Penn  State.  This  was  in  preparation  for  a 
larger  study  to  be  run  at  Quantico. 

3.1  Participants 

We  recruited  27  undergraduate  students.  Each  participant  received  $7  per  hour  for  approximately  4  to  5  hours  after 
the  study  was  completed.  Due  to  unexpected  user  behavior,  mistakes,  and  data  loss,  we  ended  up  collecting  10  sets 
of  data  (both  pre-  and  post-test  exist  and  valid)  that  we  used  for  analysis. 


3 


3.2  Materials  and  Apparatus 

All  participants  used  a  Windows  laptop  computer  with  build-in  keyboard  and  touchpad  in  a  lab  environment.  The 
D2P/MTT  program  of  version  0.5.2  (17  January  2012  release)  was  installed  and  tested  before  an  experiment  was 
started.  All  activities  were  completed  in  an  isolated,  closed  room  with  no/minimal  distraction. 

3.3  Research  Design  and  Procedure 

Participants  first  finished  a  pre-test  asking  them  to  judge  angle,  range,  speed,  and  POA  of  24  videos  showing  a 
moving  target  who  is  running  (10  mph)  or  walking  (5  mph)  at  25,  50,  or  100  meters  (range)  at  45  or  90  degree 
angles  (summary  of  questions  is  shown  in  Table  1)  without  feedback  so  that  they  did  not  know  whether  their 
answers  were  right  or  wrong.  Questions  were  presented  in  a  fixed  order  for  all  participants.  There  were  no  time 
constraints  for  both  pre-  and  post-tests — Participants,  however,  were  instructed  to  complete  both  tests  as  fast  as  they 
could. 


Table  1 .  Number  of  questions  for  each  type 


Categories 

Number  of  Questions 

Angle  (45  and  90  degree) 

4 

Range  (25,  50,  and  100  yards) 

6 

Speed  (5  and  10  mph) 

4 

Point  of  aim 

10 

Both  pre-  and  post-test  were  administrated  online  and  the  questions  were  stored  online.  Participants  used  a  web 
browser  to  complete  both  tests.  The  first  page  of  both  pre-  and  post-test  is  the  instruction  for  completing  the  test. 
Figure  4  shows  one  of  the  pre-test  questions.  A  participant  typically  reads  the  question,  clicks  “Play  Video”,  enters 
an  answer,  and  then  clicks  “Next”  to  advance  to  the  next  question.  All  key  strokes,  mouse  movements,  and  mouse 
clicks  were  recorded  using  a  keystroke  logging  application  named  RUI  (Kukreja,  Stevenson,  &  Ritter,  2006)  to 
capture  their  answers  for  data  analysis. 


According  to  the  video  and  the  image,  what  should  be  the  points  of  aim?  Play  Video 


A.  0 

B.  1 

C.  2 

D.  3 

E.  4 

r~  ~  i 


Figure  3.  An  example  of  one  pre-test  question  asking  the  POA 

After  the  pre-test,  participants  used  the  instructional  materials  in  the  D2P/MTT.  They  completed  the  training  that 
includes  practicing  exercises  with  feedback  showing  whether  their  answers  were  right  or  wrong  and  what  the  right 
answer  was  if  wrong.  The  number  and  length  of  training  sessions  varied  across  participants.  .  Participants  completed 
the  training  at  their  own  pace,  with  no  time  limits,  and  all  of  them  completed  it  within  five  hours. 
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In  the  end  of  the  training,  participants  were  tested  again  using  the  post-test.  The  processes  of  administering  post-test 
were  identical  to  the  pre-test  in  terms  of  the  format  and  the  number  of  questions  in  each  category.  Key  strokes, 
mouse  movements,  and  mouse  clicks  were  recorded  using  RUI  (Kukreja  et  ah,  2006).  The  only  difference  was  the 
question  sequence,  which  was  a  different,  fixed  order  across  participants. 


4  Results 

The  following  results  are  based  on  ten  sets  of  data  that  include  both  pre-  and  post-test  so  that  we  can  compare 
participants’  performance  before  and  after  the  MTT  training.  The  results  are  calculated  from  10  participants  because 
we  experienced  data  loss  in  the  experiment.  We  have  both  pre-  and  post-test  from  all  these  10  participants  and  lost 
either  pre-  or  post-test  or  both  of  the  other  17  participants.  This  data  loss  will  be  discussed  in  the  discussion  section. 

Table  2  presents  the  pre-test  and  post-test  scores.  Categorically,  their  performance  of  accuracy  for  target  speed, 
angle,  and  range  is  significantly  better  than  in  POA  (85%,  80%,  and  72%,  respectively,  versus  20%).  It  appears  most 
participants  can  basically  recognize  the  components  of  the  skill,  but  do  not  have  the  ability  to  retrieve  the  correct 
POA  before  the  training.  These  results  indicate  that  without  training,  our  participants  did  not  know  how  to  determine 
POA.  They  may  not  have  known  what  POA  is  and  the  concept  of  leads  before  the  training. 


Table  2.  Results  of  Pre-test  and  Post-test  Accuracy  by  Categories 


Mean 

SD 

pre-test 

post-test 

pre-test 

post¬ 

test 

t-value 

P 

All 

54% 

62% 

32% 

19% 

-0.680 

.505 

All  but  POA 

78% 

72% 

31% 

12% 

0.571 

.575 

Speed  only 

85% 

70% 

10% 

22% 

1.963 

.065 

Angle  only 

80% 

75% 

8% 

6% 

1.581 

.131 

Range  only 

72% 

72% 

18% 

8% 

0 

1.000 

POA  only 

20% 

47% 

15% 

18% 

-3.644 

.002 

By  comparing  their  POA  performance  of  pre-  and  post-tests,  we  see  an  increase  of  correct  answers  from  20%  (SD  = 
15%)  to  47%  (SD  =  18%).  It  indicates  that  using  D2P/MTT  improves  their  performance  by  1.8  standard  deviations 
in  POA  category.  This  is  a  significant  improvement  for  skill  acquisition  using  computer-based  tutoring  system 
(Bloom,  1984),  and  Table  2  shows  that  this  is  a  reliable  difference. 

To  further  understand  the  reason  and  area  of  improvement,  we  need  to  compare  the  MTT  training  with  other 
learning  conditions  such  as  the  current  POA  instructional  condition.  Also,  the  performance  of  both  speed  and  angle 
drop  slightly,  and  we  do  not  know  the  reason  without  further  analysis. 

We  also  analyze  our  data  based  on  the  subjects’  task  completion  time  on  the  pre-  and  post-test,  that  is,  the  time  from 
when  the  subject  presses  “Play  Video”  on  the  first  question  until  he  enters  the  last  answer.  Table  3  shows  that  the 
participants  are  on  average  90  s  faster  in  the  post-test  in  responding  to  the  questions.  We  also  want  to  note  that  all 
ten  participants  improved  in  their  completion  time. 


Table  3.  Descriptive  Statistics  for  POA  Accuracy  and  Completion  Time 


Pre-test  (SD) 

Post-test  (SD) 

t 

P 

POA  accuracy  (#  of  questions) 

2.10(1.66) 

4.70  (1.77) 

3.344 

0.0086 

Completion  time  (seconds) 

279.56  (47.44) 

189.71  (30.30) 

5.341 

0.00047 
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5  Discussion 

D2P/MTT  is  designed  to  teach  judging  POA  quickly.  Based  on  the  collected  data  and  results  (both  summary  and 
descriptive  statistics),  D2P/MTT  appears  to  be  effective  training  users  in  the  acquisition  of  POA.  After  the  MTT 
training  sessions  (less  than  five  hours),  the  participants  improved  in  both  accuracy  and  completion  time 
significantly.  The  accuracy  rate  (only  the  POA  score)  improved  almost  two  standard  deviations,  a  relatively  large 
effect. 

We  attribute  the  significant  improvement  in  POA  to  the  instructional  design  of  MTT  and  the  opportunity  to  practice 
to  proceduralize  the  skill.  This  improvement  may  also  be  attributed  to  the  novelty  of  our  subjects  before  the  training. 
Regular  college  students  do  not  need  to  use  POA  skills  in  their  everyday  life,  which  reflects  in  their  low  pre-test 
scores  (20%).  Further  studies  are  required  to  refine  the  results  and  to  understand  the  attribution  factors. 

During  the  study,  we  lost  10  sets  of  data,  we  lost  some  or  entire  data  sets  from  another  17  participants.  The  data  loss 
rate  is  unusually  high  (63%).  In  the  following  sections  we  discuss  things  that  can  be  improved  in  future  or  similar 
study. 

5.1  Unexpected  User  Behavior 

In  the  instruction  of  both  pre-  and  post- test  we  stated  “Do  not  change  your  answer.”  We  still  see  several  log  files  that 
have  more  answers  than  the  number  questions  in  the  RUI  log  file.  It  is  understandable  that  people  in  general  want  to 
receive  a  high  score,  even  though  they  will  not  see  the  actual  score  in  the  end.  But  changing  answers  has  made 
scoring  difficult  because  with  our  initial  software  we  could  not  completely  correlate  the  answers  with  questions.  We 
noticed  this  problem  after  we  had  collected  several  pre-tests.  We  reminded  the  rest  of  our  participants  to  make  sure 
they  were  all  very  clear  about  this  instruction.  However,  some  participants  had  been  exposed  to  the  training 
materials.  We  could  no  longer  use  their  pre-test  scores,  and  we  could  not  ask  them  to  retake  the  pre-test. 

In  addition,  we  instructed  our  participants  to  complete  the  test  “as  fast  as  you  could.”  Several  log  files  suggested  that 
the  “Play  Video”  button  had  been  clicked  more  than  once.  Again,  we  realized  this  problem  and  used  JavaScript  to 
disable  the  “Play  Video”  button  as  soon  as  it  was  clicked.  This  approach  makes  the  log  file  cleaner  and  timing 
should  be  more  realistic  to  the  purpose  of  the  MTT  training. 

To  avoid  this  problem  we  have  revised  our  software  and  instructions.  At  Quantico,  the  new  software  should  be  used. 
Pilot  logs  should  also  be  gathered  and  tested,  and  the  people  running  the  study  should  practice  and  be  practiced  in 
gathering  the  logs  and  saving  them. 

5.2  Administrative  Oversight 

The  research  assistants  (RAs)  who  ran  the  experiment  had  piloted  at  least  once  using  our  lab  members.  We  did  not 
attempt  to  check/analyze  log  files  because  the  pilot  participants  used  paper  and  pencil  to  write  down  the  answers. 

We  changed  the  protocol  to  include  the  RUI  software  keystroke  to  reduce  task  switching — from  computer  screen  to 
paper.  As  a  result,  the  problems  we  had  experienced  were  not  discovered  until  the  real  data  were  collected. 

There  were  also  changes  in  the  middle  of  the  study  that  caused  a  problem  in  the  post-test  in  which  question  number 
three  was  linked  to  question  number  thirteen.  This  results  in  only  fourteen  answers  instead  of  the  expected  twenty- 
four.  We  managed  to  ask  some  participants  to  retake  the  post-test  but  not  all  came  back.  We  end  up  having  several 
subjects  with  a  pre-test  but  without  a  corresponding  post-test  and  we  had  to  exclude  them  from  our  data  analysis. 

To  avoid  this  problem  the  logs  should  be  checked  more  often  and  the  materials  have  been  revised.  At  Quantico,  the 
logs  should  be  checked  with  pilot  subjects  and  routinely  collected  after  each  pre-  and  post-test,  and  the  logs  checked 
after  each  Marine  is  run. 


6 


5.3  Data  Loss 

For  unknown  reasons,  the  RAs  were  not  able  to  provide  some  pre-tests  and  post-tests.  We  do  not  know  whether  the 
participants  did  the  test  or  not.  We  simply  do  not  have  the  log  files.  We  also  have  zero-sized  log  files,  and  we  do  not 
know  the  reason.  This  is  the  first  experimental  study  that  the  RAs  had  run  and  the  first  we  had  run  with  multiple 
RAs,  and  the  reasons  for  the  random  lose  could  be  attributed  to  careless  or  novelty  of  the  RAs  when  they  ran  the 
study,  or  our  lack  of  detailed  oversight. 

To  avoid  this  problem  we  will  check  the  RAs  checking  the  logs.  At  Quantico,  the  logs  should  be  routinely  collected 
after  each  pre-  and  post-test,  and  the  logs  checked  after  each  Marine  is  run. 


6  Conclusions  and  Future  Study 

This  is  the  first  experiment  ran  for  the  MTT.  We  want  to  know  whether  it  is  effective  and  useful  the  results  confirm 
that  the  MTT  is  capable  of  what  we  had  hoped  for.  The  after  training  score  (47%  accuracy  for  POA)  is  not  as  high  as 
we  would  like,  and  the  ACS  Lab  is  working  on  improving  and  polishing  the  instruction  to  ensure  that  the  accuracy 
rate  can  be  higher  after  the  MTT  training.  The  new  and  improved  the  MTT  should  be  reevaluated  when  it  is 
completed.  This  score  might  also  be  increased  by  providing  additional  time  with  the  tutor. 

This  study  does  not  confirm  or  deny  that  the  skill  acquisition  moved  from  declarative  to  procedural  for  our 
participants  through  an  entire  MTT  training.  It  takes  other  studies  with  different  designs  to  test  this  statement.  This 
work  also  required  more  knowledge  about  the  retrieval  time  for  declarative  information  as  well  as  procedural 
knowledge.  But,  we  know  that  the  current  tutor  made  learners  are  more  accurate  retrieving  POA  and  faster  at 
making  the  component  judgments  and  retreieving  the  correct  point  of  aim. 
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